Knowledge Spaces for Inductive Reasoning Tests

Die Wissensraumtheorie (Doignon & Falmagne, 1985;1999), in der Testitems durch sog. Vermutungsrelationen geordnet werden, wurde verallgemeinert, um Vermutungsrelationen zwischen Tests zu definieren (Albert, 1995, Brandt, Albert & Hockemeyer, 1999;accepted). Als erste exemplarische Anwendung dieses Modells werden verschiedene Typen von induktiven Denkaufgaben nach dem Prinzip der komponentenweisen Ordnung strukturiert und durch Vermutungsrelationen in Beziehung gesetzt. Die Hypothesen konnten durch die Validierung der resultierenden Aufgabenund Teststrukturen an einem reanalysierten Datensatz weitgehend bestätigt werden. Doignon and Falmagne (1985;1999) developed the behavioral theory of knowledge spaces for the efficient assessment of knowledge. A student’s knowledge is described by his or her knowledge state, i.e. the set of items this student is able to solve. The items are ordered partially by defining a surmise relation on the set of items. The interpretation of a surmise relationship (or prerequisite relationship) between two items x and y is that from the correct solution of item x, we can surmise the correct solution of item y (item y is prerequisite for item x). Thus, the possible combinations are reduced to the set of knowledge states, called knowledge space (K). Albert (1995), Brandt, Albert, and Hockemeyer (1999; accepted) extended the concept to a set of tests T, i.e. to surmise relations between subsets of items. The interpretation of a prerequisite or surmise relation between tests (BSA) is that two tests A, B ∈ T are in surmise relationship from A to B, if from the correct solution of at least one item a ∈ A we can surmise the correct solution of a particular subset of item(s) b ∈ B. More specific types of surmise relations between tests are the left-covering (BSlA, each item a ∈ A has a prerequisite b ∈ B), right-covering (BSrA, each item b ∈ B is prerequisite for an item a ∈ A), and total-covering (BStA, leftand right-covering) surmise relation. The interpretation of a left-covering surmise relation is that a person who doesn't solve any item in test B will not be able to solve any item in test A, either. There is no need to test this person on test A. Right-covering, on the other hand, means that a person who solves all items in test A is also able to solve all items in test B. Hence, there is no further need to test this person on test B. Advantages of the concept of surmise relations between tests are the possibility to specify prerequisite relationships between entire tests, cognitive, or developmental stages and that adaptive testing systems, covering a wide range of item classes, can be developed for more economical testing procedures. ∗ This work was supported by the Austrian Science Foundation (FWF) under the Grant P12726SOZ to Dietrich Albert and Wilhelm Schappacher. We also want to thank the “Heerespsychologischer Dienst des BMLV” in Vienna for the provision of data. Gudrun Wesiak and Dietrich Albert 2 Component based ordering of inductive reasoning tests For surmise relations between items (within tests) several methods and principles have been used to structure a set of items (see Albert & Lukas, 1999). According to the principle of componentwise ordering of product sets the set of items is described by common components. Furthermore, each component consists of a set of attributes, on which a difficulty order is defined. The Cartesian Product of the components results in the set of attribute combinations, which corresponds to the set of possible item classes. The surmise relation on items is established through a pairwise comparison with respect to the components' attributes. Furthermore, a partial order can be defined on the components, meaning that components are assumed to be independent or of different importance. For the establishment of a surmise relation between tests, items of different tests have to be analysed and described by components and attributes which are applicable to the item classes. This way, items of different tests can be compared and surmise relations between the tests can be established. As a first step to construct a test knowledge structure for inductive reasoning tests (analogies, classifications, matrices, and series continuations), components and attributes were defined. Included are the components material, number and difficulty of operations (induced rules for solving a problem), task ambiguity (similarity of answer alternatives), salience (difficulty to detect the relevant rule), and number of answer alternatives. In a second step the attributes of each component were ordered according to their difficulty. The assumed orders are, among others, based on findings by Albert and Held (1999), Albert and Musch (1996), Holzmann, Pellegrino, and Glaser (1983), Klauer (1993), Schrepp (1999), and Sternberg and Gardner (1983). Table 1: Components, attributes, and difficulty orders for inductive reasoning item classes Components Attributes Difficulty orders A: Answer alternatives ai = 2, 3, ..., n {a< i} B: Operation difficulty* b1 O other (less difficult) { } b2 X difficult {b1} C: Number of operations di = 1 4 {d<i} D: Task Ambiguity d1 L low ambiguity { } d2 H high ambiguity {d1} E: Salience e1 L low difficulty { } e2 H high difficulty {e1} F: Material f1 P – pictorial { } f2 L letters {f1} f3 V verbal {f1} f4 N numerical {f1,f2} f5 G geometric-figural {f1,f2,f3,f4} Note. *Types of operations (B) vary with respect to component F (e.g. rotation is only applicable to geometric, class inclusion to verbal material). Table 1 shows the specified components, their respective attributes, and the assumed difficulty orders for each component. Finally, components were orInductive Reasoning and Knowledge Spaces 3 dered by importance and the resulting item classes were ordered by forming the components' product and comparing their attributes’ difficulty. From the obtained knowledge structure surmise relations within tests (i.e. for the items of each test, e.g. analogies) and between tests (analogies, classifications, etc.) have been derived. Empirical Application and Results For a first evaluation of the established structure, we reanalysed a subset of data (809 subjects) for two inductive reasoning tests (15 verbal analogies and 12 geometric matrices) developed by the “Heerespsychologischer Dienst des BMLV” in Vienna. The items were presented in paper-pencil form (group sessions) and subjects had to choose between eight alternatives for the matrix test and five alternatives for the analogy test. To establish a knowledge structure for the two tests, the items were analysed according to the components and attributes mentioned above (e.g. item class 5O1LLV has 5 answer alternatives (A), Other operation difficulty (B), 1 operation (C), Low ambiguity (D), Low demand (E), and Verbal material (F)). Difficulty orders on the attributes correspond to column three in Table 1, components were ordered partially by defining components B, C, D, and E as being more important than components A and F. The resulting hypotheses on the surmise relations between the two tests are that there is a right-covering surmise relation from the analogy test to the matrix test (MT Sr AN each matrix item is prerequisite for an analogy item) and a left-covering surmise relation from the matrices to the analogies (AN Sl MT each matrix item has a prerequisite in the analogy test). Figure 1 illustrates a simplified substructure between the tests. Figure 1: Hypothesised surmise relations between relevant parts of the matrix and the analogy test (relationships that can be inferred through transitivity are not shown) For the validation of the hypotheses the fit of the surmise relation to the set of data was estimated by calculating the relative solution frequencies for each item class. Regarding the percentage of correct solutions the overall rule for the interpretation of solution frequencies is that whenever an item y is prerequisite for another item x, item y should have a higher solution frequency than item x. The results for the matrix test showed a very good fit of hypothesised and empirical structure, while minor deviations occurred within the analogy Gudrun Wesiak and Dietrich Albert 4 test (item classes 5O1HLV and 5O1LHV were classified as too easy). Regarding the relationships between item classes of different tests 16 out of 27 pairs confirm the left-covering, and 25 out of 28 pairs the right-covering surmise relation. However, the overall hypotheses of a left-covering surmise relation from the matrix test to the analogy test and a right-covering surmise relation from the analogy test to the matrix test could be verified by seven out of eight item classes each (the matrix item class 8O1LLG does not have a prerequisite in the analogy test and item class 8O3LHG is not prerequisite for an analogy item class).